This article introduces the Gemma 4 family of models from Google DeepMind, detailing their architectures and improvements over the previous version, Gemma 3. It highlights key features such as interleaved attention layers and efficiency enhancements in global attention mechanisms.

• Gemma 4 comes in four sizes—E2B, E4B, 31B, and a 26B MoE model that only activates 4B parameters at inference—and handles text, images, and audio
• Local-to-global attention layers are interleaved at a fixed 4:1 ratio in smaller models and 5:1 in larger ones, with sliding windows of 512 tokens (E2B/E4B) or 1024 tokens (larger models) to cut compute costs
• Global attention memory use is reduced through Grouped Query Attention, a K=V trick that makes keys and values identical, and p-RoPE for improved positional encoding